Papers with pretraining methods
Structure-Grounded Pretraining for Text-to-SQL (2021.naacl-main)
Copied to clipboard
Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, Matthew Richardson
| Challenge: | STRUG is a weakly supervised structure-based pretraining framework for text-to-SQL . it can be used to learn to capture text-table alignment in a given database schema . |
| Approach: | They propose a weakly supervised structure-grounded pretraining framework for text-to-SQL that can effectively learn to capture text-table alignment based on a parallel text-tab corpus. |
| Outcome: | The proposed framework outperforms BERTLARGE and BERTLAGE on all text-to-SQL alignment settings. |
How much pretraining data do language models need to learn syntax? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining methods are convenient, but expensive in terms of time and resources. |
| Approach: | They investigate the impact of pretraining data size on the syntactic capabilities of RoBERTa by using syntaktic structural probes to determine whether models pretrained on more data encode a higher amount of syntastic information. |
| Outcome: | The proposed models perform better on part-of-speech tagging, dependency parsing and paraphrase identification. |
Multi-pretraining for Large-scale Text Classification (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for large-scale text classification involve excessive computation and memory overheads. |
| Approach: | They propose a self-supervised and weakly supervised pretraining frameworks for large-scale text classification with multiple categories. |
| Outcome: | The proposed framework improves on the self-supervised and weakly supervised methods while being computationally efficient. |
mDAPT: Multilingual Domain Adaptive Pretraining in a Single Model (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing domain-specific multilingual pretraining data is difficult to obtain due to regulations, legislation, or simply a lack of language- and domain- specific text. |
| Approach: | They propose to continue pretraining a language model on domain-specific unlabelled text . this allows for better modelling of text for downstream tasks within the domain . |
| Outcome: | The proposed approach outperforms the general multilingual model and performs close to its monolingual counterpart. |
Patton: Language Model Pretraining on Text-Rich Networks (2023.acl-long)
Copied to clipboard
| Challenge: | Existing models for text-rich networks do not take inter-document structure into account. |
| Approach: | They propose a pretraining framework for a text-rich network using a masked language model and a masking node prediction framework. |
| Outcome: | The proposed model outperforms baselines on four tasks in academic and e-commerce domains. |
Structure-Aware Language Model Pretraining Improves Dense Retrieval on Structured Data (2023.findings-acl)
Copied to clipboard
| Challenge: | Structure Aware Dense Retrieval (SANTA) model encodes user queries and structured data in one universal embedding space for retrieving structured data. |
| Approach: | They propose to use structured data and unstructured data to encode queries and structured data in one universal embedding space for retrieving structured data. |
| Outcome: | The proposed model achieves state-of-the-art on code search and product search and conducts convincing results in the zero-shot setting. |
Mitigating Label Biases for In-context Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to categorize label biases in in-context learning (ICL) have not addressed all three types of label bias. |
| Approach: | They propose a method that estimates a language model’s label bias using random in-domain words from the task corpus to categorize and detect label biases in ICL. |
| Outcome: | The proposed method significantly improves the performance of GPT-J and GPT-3 on a wide range of tasks. |
SCROLLS: Standardized CompaRison Over Long Language Sequences (2022.emnlp-main)
Copied to clipboard
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, Omer Levy
| Challenge: | Standard NLP benchmarks focus on short texts, but long texts are produced in the context of longer discourses. |
| Approach: | They propose a new benchmark that places models in context of long texts that require reasoning over long texts. |
| Outcome: | The proposed task sets are based on a set of long-text datasets and host a live leaderboard to facilitate research on model architecture and pretraining methods. |
DocSplit: Simple Contrastive Pretraining for Large Document Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing model pretraining methods only consider local information, resulting in low-quality embeddings for large documents. |
| Approach: | They propose a new method which forces models to consider the entire global context of a large document. |
| Outcome: | The proposed method outperforms existing models on document classification, few shot learning, and retrieval tasks. |
Meta-Adapter for Self-Supervised Speech Models: A Solution to Low-Resource Speech Recognition Challenges (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing self-supervised learning models can learn latent representations from large amounts of unlabeled data, but they are expensive to fine-tune. |
| Approach: | They develop a meta-adapter to obtain meta-initialized parameters for self-supervised models . meta-Adapters show better generalization and extensibility than traditional pretraining methods . |
| Outcome: | Experiments on common voice and FLEURS datasets show Meta-Adapter performs better on low-resource languages . authors show it can be used on 12 low-source languages, but it requires huge computational resources . |